Development of a national proficiency test for SARS-CoV-2 detection by PCR in Colombia

Background Proficiency testing (PT) is a tool for ensuring the validity of results of testing laboratories and is essential when laboratories are working with assays authorised for emergency use or implementing novel techniques for detecting emerging pathogens. Methods In collaboration with the National Health Institute of Colombia and with international support, we developed a qualitative PT for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) by reverse transcription polymerase chain reaction (RT-PCR). A proficiency test item (PTI) based on reference material (research grade) produced by the National Institute of Standards and Technologies (NIST) was prepared and characterised using three positive samples with varying concentrations of SARS-CoV-2 ribonucleic acid (RNA) and two negative (control) samples. Tests were distributed to 121 laboratories across the national network of public health laboratories in Colombia. Results Positive samples had varying concentrations of SARS-CoV-2 RNA and were quantified by digital PCR (RT-ddPCR) assays for the E gene of SARS-CoV-2. We tested the ability of laboratories to detect low and high levels of viral RNA using samples with SARS-CoV-2 RNA concentrations of 1417 ± 216, 146 ± 28, and 14 ± 10 copies /uL (expanded uncertainty, k = 2, 95% confidence level) We also performed a semiquantitative analysis of instrumental responses (Ct values) reported by participating laboratories and homogeneity, stability, and characterisation studies of the produced materials to determine the adequacy of these materials and methods for use in the qualitative PT scheme. The PT evaluated reports for individual target genes from each laboratory; 98.3% of laboratories had satisfactory performance and the remaining 1.7% of laboratories had unsatisfactory performance for the detection of at least one of the reported genes. Conclusions This PT scheme identified the potential metrological weaknesses of laboratories in the detection of SARS-CoV-2 by RT-PCR and may facilitate improvements in the quality of measurements from the perspective of public health surveillance.

Once the RT-PCR methods had been optimized (PCR efficiency, annealing temperature, reaction time, and heating ramp), the selected performance parameters were evaluated for each target gene (Table S2).The working interval for each PCR method was evaluated by linear regression analysis using ordinary least squares between the working solution dilutions versus the instrumental response (for qPCR) and the concentration response (for dPCR) in the selected ranges.The acceptance criteria was a slope statistically different from zero and a correlation coefficient >0.99.Precision was evaluated using analysis of variance with the inter-day difference used as the variation factor.A relative standard deviation of <25% was set as the acceptance criteria.The LoQ was defined as the minimum concentration level of the working interval.

qPCR validation
Serial 10-fold gravimetric dilutions prepared from the working solution were at concentrations ranging from 0.2 to 200,000 copies/μL, To determine the LoD of RT-qPCR, serial gravimetric dilutions from 6 to 0.2 copies/μL were measured in four replicates.The LoD was defined as the minimum concentration where four replicates were positive.

dPCR validation
Gravimetric dilutions were prepared from the working solution at a range of 4500 to 0.2 copies/μL to evaluate the performance selected parameters (linearity, repeatability, inter-day intermediate precision, LoD, and LoQ), For RT-dPCR, serial gravimetric dilutions from 0.5 to 0.04 copies/μL were used in triplicate.The LoD was defined as the lowest concentration with a minimum of 9 positive partitions in three replicates [11].Table S2 summarizes the results for each validation parameter.Regarding the validation of PCR methods required for PTI characterization, ddPCR, has a higher dispersion for low concentration samples, and has a lower LoD compared to the qPCR method.
Specificity has previously been evaluated using the reference methods (Table S1).The repeatability of the ddPCR method at low concentrations contributed to higher uncertainty during PTI characterization.
Both qPCR and ddPCR methods have satisfactory performance in measuring SARS-CoV-2 RNA copy number (copies/μL) in samples containing 10 to 1000 copies/uL (Table S2).Defined concentrations for reference material.

Text S3: Pilot Study: Proficiency Test Item
Prior to PTI production, a pilot study was performed as a prospective study to evaluate the concentration levels, homogeneity, and stability of the materials at −20°C and 4°C, with −70°C used as a reference temperature (Figure S2).Additionally, as some laboratories performed PT with closed platforms (automated RNA extraction, purification, and RT-PCR), three dilutions were prepared from a 10 copies/μL SARS-CoV-2 RNA solution.
The results of the pilot study determined the PTI preparation conditions (dilutions, volumes, nominal concentrations, buffers, and vial materials) required to ensure the proposed PTI was suitable for open and closed platforms (Table S3).Based on the results of this study, a reference material for the PTI was developed.Homogeneity, stability, and characterization studies were performed to demonstrate the PTI was fit for purpose.

Homogeneity study
The heterogeneity of the material was evaluated using one-factor ANOVA.The uncertainty component for homogeneity (uhom) was calculated according to Eq 1.
where ubb is the uncertainty associated with between-unit variability that corresponds to sbb (the standard deviation of between-unit component from a homogeneity study), MSbetween is the between-group mean square, MSwithin is the within-group mean square or repeatability variance, and n is the number of replicates.
When MSwithin is greater than MSbetween, the sbb was calculated according to Eq 2. [9] Where   ℎ are the degrees of freedom of the MSwithin.

Stability study
The uncertainty contribution due to stability was calculated from regression analysis as the slope standard deviation and the time in weeks (Eq 3).The relative uncertainties were calculated using the concentration value as a reference.

Value assignment
The previously validated RT-dPCR E-Charité assay was used under repeatable conditions in triplicates to determine the mean of five selected vials per concentration level ( ℎ ); where the components associated with homogeneity ( ℎ ) and stability (  were zero (Eq 4).
=  ℎ +  ℎ +    4 The combined standard uncertainty of the reference value was calculated according to Eq 5, Where uchar comes from the mathematical model that describes the copy number concentration measured (  ) combined with the measurement precision (R; Eq 6 and 7)

Text S7: Proficiency Test Results, Control Use Report
From the participating laboratories, 101 reported the use of positive controls.For commercial kits and closed platforms, 72.1% of laboratories used the positive control included in the commercial kit used.For in-house protocols, 12.6% of laboratories used controls that had been developed in-house and 6.3% of laboratories used controls acquired from an external supplier or one that differed from the kit manufacturer (NIST, lNS, or manufacturers such as SeraCare, Eurofins, or Vircell; Figure S3 blue bars).
Regarding negative controls, participating laboratories used a range of materials; 46.8% used the negative control included in the commercial kit used, 29.7% used nuclease-free water, 1.8% used controls from an external supplier or one that differed from the kit manufacturer, and 9.0% used no template control (Figure 5, red bars).The LoD values reported by participating laboratories, based on the information provided in the RT-PCR kits used, values obtained during the method verification, or reported by previous scientific literature, included a wide variety of values and units (copies/μL, copies/mL, copies/reaction, genomic equivalents/mL, μM, ng/μL, and Ct values, among others; Table S9).Most laboratories used 5 μL of the PTI solution (Figure S4), equivalent to 70 copies per reaction.Laboratories reported instrumental response values from one, two, or three replicates depending on the protocol and assay used (Table S10).

Figure S8. 1 .
Figure S8.1.Volumes of the PTI used for RT-PCR reactions using open platforms.

Table S2 .
Summarized results for RT-PCR validation methods.
a lowest concentration corresponding to the linear range for the N assay.bRange of maximum precision values in terms of repeatability for the highest and lowest concentrations in the linear range corresponding to the N (CDC) and E INM assays.Intermediate precision values corresponds to the RdRp assay only."

Table S3 . 1
Results from the pilot study of materials used in closed platforms.
*Results in parentheses correspond to CT values for ORF1ab and E genes.

Table S5 . 1 .
Regression analysis for short-and long-term stability study for PTI

Table S5 .
2 ddPCR raw data for value assignment for low, medium, and high concentration samples of PTI.

Table S5 . 3
Analysis of variance of results of table S1 dataAs an informative value, RdRp copy number was measured in the three positive samples in one panel in triplicate using RT-ddPCR to check the assigned value in the second fragment of the RGTM 10169 NIST material.
1 Expanded uncertainty at k = 2.1 for a 95% confidence level.TextS6: Proficiency Test Platform ResultsTableS6.1Platformsand reagents information reported by laboratories.

Table S8 . 1 .
Range and frequency of limit of detection values reported by laboratories.Laboratories reporting only copy numbers or copy number/test were classified as copies/reaction. *

Table S8 . 2 .
Proportions of laboratories reporting instrumental response values from one, two, or three replicates according to the gene tested.